Area_based_analysis
Brandon Titensor
2025-01-08
1. Data Loading and Preprocessing
1.1 Edge Data
1.2 Surface Data
1.3 Calibration Wafer Data
1.4 Calibration Surface Data
1.5 Witness Surface Data
2. Surface Area Distribution Analysis
3. Edge Area Distribution Analysis
4. Calibration Wafer Analysis - Edge
5. Calibration Surface Analysis
6. Witness Surface Analysis
6. Edge Model Analysis
7. Straight Line Model (SLSM)
## witness_surface_before9: 701 observations
## witness_surface_before10: 887 observations
## witness_surface_before11: 1295 observations
## witness_surface_before12: 214 observations
## witness_surface_before13: 1066 observations
## witness_surface_before14: 204 observations
## witness_surface_after9: 292467 observations
## witness_surface_after10: 144961 observations
## witness_surface_after11: 224730 observations
## witness_surface_after12: 121144 observations
## witness_surface_after13: 58732 observations
## witness_surface_after14: 99428 observations##7.5
## === BRIGHTNESS ANALYSIS INITIALIZATION ===## Analyzing image brightness patterns across surface and edge types...## Target analyses: Starshade Surface, Calibration Surface, Witness Surface, Starshade Edge, Calibration Edge## === SURFACE BRIGHTNESS ANALYSIS ===
## Available surface datasets: Starshade Surface Before, Starshade Surface After, Calibration Surface Before, Calibration Surface After, Witness Surface Before, Witness Surface After
## Processing Starshade Surface Before - Total images: 15111
## Processing Starshade Surface After - Total images: 15120
## Processing Calibration Surface Before - Total images: 6146
## Processing Calibration Surface After - Total images: 7631
## Processing Witness Surface Before - Total images: 1413
## Processing Witness Surface After - Total images: 3010
## Total surface images analyzed: 48431
## Images with transformation applied: 35462
## # A tibble: 6 × 4
## Surface_Type Timing Total_Images Transformed_Images
## <chr> <chr> <int> <int>
## 1 Calibration After 7631 7631
## 2 Calibration Before 6146 6146
## 3 Starshade After 15120 10547
## 4 Starshade Before 15111 8263
## 5 Witness After 3010 2006
## 6 Witness Before 1413 869## === EDGE BRIGHTNESS ANALYSIS ===
## Available edge datasets: Starshade Edge, Calibration Edge
## Processing Starshade Edge - Total samples: 975
## Extracting trial numbers from column: image_name
## Available brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Using brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median## Processing Calibration Edge - Total samples: 994
## Extracting trial numbers from column: image_name
## Available brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Using brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median## Total edge measurements analyzed: 23628
## # A tibble: 2 × 3
## Edge_Type Total_Samples Total_Measurements
## <chr> <int> <int>
## 1 Calibration 994 11928
## 2 Starshade 975 11700## Creating surface brightness visualizations...## Attempting timing effect analysis...
## Columns created after pivot_wider: Surface_Type, Trial
## Expected columns found:
## Could not find expected columns for timing effect analysis.
## Available columns: Surface_Type, Trial
## Creating edge brightness visualizations...
##
## === SURFACE BRIGHTNESS SUMMARY STATISTICS ===
## # A tibble: 6 × 8
## Surface_Type Timing Count Original_Mean Original_SD PostTransform_Mean
## <chr> <chr> <int> <dbl> <dbl> <dbl>
## 1 Calibration After 7631 3.8 2.7 3.4
## 2 Calibration Before 6146 4.8 2.2 3.5
## 3 Starshade After 15120 2.1 1.6 2
## 4 Starshade Before 15111 1.6 1.6 1.5
## 5 Witness After 3010 2 1.4 2
## 6 Witness Before 1413 1.8 1.5 1.8
## # ℹ 2 more variables: PostTransform_SD <dbl>, Avg_Change_Percent <dbl>
##
## === SURFACE BRIGHTNESS NORMALIZATION EFFECTIVENESS ===
## # A tibble: 6 × 5
## Surface_Type Timing Original_CV PostTransform_CV CV_Improvement_Percent
## <chr> <chr> <dbl> <dbl> <dbl>
## 1 Calibration After 0.708 0.626 11.5
## 2 Calibration Before 0.463 0.632 -36.5
## 3 Starshade After 0.748 0.751 -0.5
## 4 Starshade Before 1.01 1.01 -0.2
## 5 Witness After 0.709 0.709 0
## 6 Witness Before 0.791 0.791 0##
## === EDGE BRIGHTNESS SUMMARY STATISTICS ===
## # A tibble: 24 × 8
## Edge_Type Contamination Processing Region Count Mean_Brightness SD_Brightness
## <chr> <chr> <chr> <chr> <int> <dbl> <dbl>
## 1 Calibrat… After Contam… After Pro… Entire 994 82.4 78.9
## 2 Calibrat… After Contam… Before Pr… Entire 994 93.6 89.1
## 3 Calibrat… Before Conta… After Pro… Entire 994 154. 60.3
## 4 Calibrat… Before Conta… Before Pr… Entire 994 170. 65
## 5 Calibrat… After Contam… After Pro… Lower 994 8 0
## 6 Calibrat… After Contam… Before Pr… Lower 994 10.5 2
## 7 Calibrat… Before Conta… After Pro… Lower 994 8 0
## 8 Calibrat… Before Conta… Before Pr… Lower 994 10.5 0.8
## 9 Calibrat… After Contam… After Pro… Upper 994 186 0
## 10 Calibrat… After Contam… Before Pr… Upper 994 206. 14.9
## # ℹ 14 more rows
## # ℹ 1 more variable: CV <dbl>
##
## === EDGE BRIGHTNESS CHANGE EFFECTS BY EDGE TYPE ===
## Average brightness change due to contamination:
## # A tibble: 12 × 5
## Edge_Type Processing Region Mean_Contamination_E…¹ SD_Contamination_Eff…²
## <chr> <chr> <chr> <dbl> <dbl>
## 1 Calibration After Proce… Entire -72.1 NA
## 2 Calibration After Proce… Lower 0 NA
## 3 Calibration After Proce… Upper 0 NA
## 4 Calibration Before Proc… Entire -76.1 NA
## 5 Calibration Before Proc… Lower 0 NA
## 6 Calibration Before Proc… Upper 0.9 NA
## 7 Starshade After Proce… Entire -3.6 NA
## 8 Starshade After Proce… Lower 0 NA
## 9 Starshade After Proce… Upper 0 NA
## 10 Starshade Before Proc… Entire -0.7 NA
## 11 Starshade Before Proc… Lower 1.1 NA
## 12 Starshade Before Proc… Upper 5.8 NA
## # ℹ abbreviated names: ¹​Mean_Contamination_Effect, ²​SD_Contamination_Effect
##
## Average brightness change due to processing:
## # A tibble: 12 × 5
## Edge_Type Contamination Region Mean_Processing_Effect SD_Processing_Effect
## <chr> <chr> <chr> <dbl> <dbl>
## 1 Calibration After Contami… Entire -11.3 NA
## 2 Calibration After Contami… Lower -2.5 NA
## 3 Calibration After Contami… Upper -20.2 NA
## 4 Calibration Before Contam… Entire -15.2 NA
## 5 Calibration Before Contam… Lower -2.5 NA
## 6 Calibration Before Contam… Upper -19.2 NA
## 7 Starshade After Contami… Entire -3.4 NA
## 8 Starshade After Contami… Lower -3.8 NA
## 9 Starshade After Contami… Upper -5 NA
## 10 Starshade Before Contam… Entire -0.5 NA
## 11 Starshade Before Contam… Lower -2.6 NA
## 12 Starshade Before Contam… Upper 0.8 NA
##
## === STARSHADE vs CALIBRATION EDGE COMPARISON ===
## # A tibble: 12 × 7
## Contamination Processing Region Calibration Starshade Difference
## <chr> <chr> <chr> <dbl> <dbl> <dbl>
## 1 After Contamination After Processing Entire 82.4 119. 37
## 2 After Contamination After Processing Lower 8 10 2
## 3 After Contamination After Processing Upper 186 200 14
## 4 After Contamination Before Processi… Entire 93.6 123. 29.1
## 5 After Contamination Before Processi… Lower 10.5 13.8 3.3
## 6 After Contamination Before Processi… Upper 206. 205. -1.1
## 7 Before Contamination After Processing Entire 154. 123. -31.6
## 8 Before Contamination After Processing Lower 8 10 2
## 9 Before Contamination After Processing Upper 186 200 14
## 10 Before Contamination Before Processi… Entire 170. 123. -46.3
## 11 Before Contamination Before Processi… Lower 10.5 12.6 2.1
## 12 Before Contamination Before Processi… Upper 205. 199. -6
## # ℹ 1 more variable: Percent_Difference <dbl>##
## === BRIGHTNESS ANALYSIS COMPLETE ===## Results stored in 'brightness_analysis_results' list for integration with main analysis.## Surface types analyzed: Starshade Before, Starshade After, Calibration Before, Calibration After, Witness Before, Witness After## Edge types analyzed: Starshade, Calibration##
## Available result components:## - surface_data: Combined surface brightness data (Starshade, Calibration, Witness × Before/After)## - edge_data: Combined edge brightness data (Starshade, Calibration)## - surface_summary: Summary statistics for all surface types and timing## - edge_summary: Summary statistics for all edge types## - normalization_effectiveness: CV improvement metrics by surface type and timing## - contamination_effects: Brightness changes due to contamination by edge type## - processing_effects: Brightness changes due to processing by edge type## - edge_comparison: Direct Starshade vs Calibration edge comparison## - surface_plots: Surface visualization plots (distribution, change_by_transform, before_vs_after, variability, processing_effect, timing_effect, comparison)## - edge_plots: Edge visualization plots (brightness_by_status, contamination_effect, processing_effect, edge_comparison)8. Error Stats
8. Trial Anlysis Graphs
8.1 Trial 9
8.2 Trial 10
8.2 Trial 10 with Error Bars
8.3 Trial 11
8.4 Trial 12
8.5 Trial 13
8.6 Trial 14
9. Average Graphs
9.1 Average of Trials
9.2 Average Air of Trials
9.3 Scatter Graphs
9.4 Scatter Air Graphs
10. Comparison Graphics
10.1 Graph Comparison Graphic Count
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.5265]
## 2 2 (2-2,1-1) arrange gtable[arrange]10.2 Graph Comparison Graphic Scatter
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.6392]
## 2 2 (2-2,1-1) arrange gtable[arrange]10.3 Graph Comparison Graphic - Normalized Count
##
## === MISSING DATA LOG ===## Total missing data entries: 0## No missing data detected - all datasets loaded successfully!##
## ========================
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.7504]
## 2 2 (2-2,1-1) arrange gtable[arrange]10.4 Normalized Scatter Graph
## Summary of missing data:
## Trial 9 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 9 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 9 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 10 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 10 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 10 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 11 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 11 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 11 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 12 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 12 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 12 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 13 - Surface Cumulative Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 13 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 13 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 13 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 14 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 14 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 14 - Edge Binned Scatter - ALL: No data lines available - using fallback## Plot list lengths:## Surface cumulative: 8## Surface binned: 8## Edge cumulative: 8## Edge binned: 8## Required length: 8
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.8630]
## 2 2 (2-2,1-1) arrange gtable[arrange]
## Grid created successfully!11. Combined Analysis and Comparison
11.1 Cumulative Comparison
11.2 Binned Comparison
12. Multiple Line analysis
Instrument signature
## [1] "=== INSTRUMENT SIGNATURE GAP PREDICTION ==="## [1] "Scale: 0.208333 microns/pixel"## [1] "Area conversion: 0.043403 microns²/pixel²"## [1] "\n=== SUMMARY COMPARISON ==="## [1] "Raw Gap Counts (before merging nearby gaps):"## Scenario Threshold Circle_Gaps Bin_Gaps Threshold_Gaps Scale_Gaps
## 1 Starshade Edge 60 46 3 7 8
## 2 Calibration Edge 40 46 3 7 8
## Raw_Total
## 1 64
## 2 64## [1] "\nMerged Gap Counts (after combining gaps within 0.05 log units):"## Scenario Threshold Merged_Total Reduction Percent_Reduction
## 1 Starshade Edge 60 27 37 57.8
## 2 Calibration Edge 40 26 38 59.4## [1] "\nGaps unique to Starshade Edge: 1"## [1] "Gaps unique to Calibration Edge: 0"## [1] "Starshade-specific gaps (first 5):"
## [1] 1.073## [1] "\n=== ANALYSIS OF OBSERVED GAP ==="## [1] "Observed gap at log10(area) = 2.3"## [1] "Area: 199.5 μm²"## [1] "Diameter: 15.9 μm"## [1] "Closest Starshade prediction: 0.188 log units away"## [1] "Closest Calibration prediction: 0.188 log units away"## [1] "*** Starshade Edge model better predicts your observed gap ***"## [1] "\n=== ARRAYS FOR RMARKDOWN ==="## # Starshade Edge Predicted Gaps (Threshold 60)## starshade_predicted_gaps <- c( 0.038, 0.092, 0.214, 0.339, 0.450, 0.533, 0.612, 0.691, 0.758, 0.884, 0.941, 0.992, 1.073, 1.155, 1.274, 1.328, 1.395, 1.457, 1.515, 1.569, 1.659, 1.736, 1.819, 1.895, 1.989, 2.081, 2.488 )## # Calibration Edge Predicted Gaps (Threshold 40)## calibration_predicted_gaps <- c( 0.038, 0.092, 0.214, 0.339, 0.436, 0.538, 0.612, 0.691, 0.758, 0.884, 0.941, 0.992, 1.155, 1.274, 1.328, 1.395, 1.457, 1.515, 1.569, 1.645, 1.747, 1.819, 1.895, 1.989, 2.081, 2.488 )## [1] "\n=== VALIDATION READY ==="## [1] "Use validate_gaps_simple() with your edge particle data:"## [1] "starshade_validation <- validate_gaps_simple(edge_particles_data, 'Area', starshade_predicted_gaps)"## [1] "calibration_validation <- validate_gaps_simple(calibration_particles_data, 'Area', calibration_predicted_gaps)"13. Paper Comparisons
## === CHECKING AVAILABLE DATASETS ===## Available datasets:## - surface_binned : TRUE
## - edge_binned : TRUE
## - cw_edge_binned : TRUE
## - model_edge_binned : TRUE## Creating surface distribution plot...
## Surface distribution plot created successfully## Creating edge distribution plot...
## Edge distribution plot created successfully## Creating normalized distribution plot...
## Normalized distribution plot created successfully## Creating ratio analysis plot...
## Ratio analysis plot created successfully## === ANALYSIS SUMMARY ===## Surface Distribution Analysis:
## - PCL: 175 μm²
## - Slope: -1.54
## - Data points: 100## Edge Distribution Analysis:
## - Total particles: 259
## - Area range: 1 - 5987 μm²
## - Data points: 100## Ratio Analysis:
## - Mean etched/calibration ratio: 2.56
## - Median ratio: 0.71
## - Maximum ratio: 31.48## Plots successfully created: 4 out of 4## ========================
